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ABSTRACT , . . .« 

• * Inferences made in integrative research reviews' are v 
as important to the validity^of behavioral science^knowi4dge i as are 
those * * * ~* * # # 
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conceptually arid operationally. rf In the data collection s,tage, an 
inquirer must decide on the population of elements that will be the 
•referent for the inquiry. Critical judgments about the Quality *f 
'data points occur, during data ^eyaluatioa.* Data collected by the j 
researcher are synthesized into a unified statement during analysis' 
and irtterpfeta&iqn. *P?esentation of the review in a public document 
is\the final stage*. One^stiidy asked reviewers* to integrate 
literatures that- vary T&fc findings apd operational homogeneity of 




^..Jin^s, and the reviewer's analytic interpretation strategy. The 
dependent variables itt both studies were reviewer perceptions about 
the tested hypotheses and recommendations for future research. 
( DWH) ^ / . ' . \ . v * * i ** r ° 
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The inferences made in integrative research reviews are a^witral to 
the' validity, of behavioral science knowledge as those i nf eren/ce,sNnade in pri- » 
mary research. , Therefore, research reviewers must pay the same, attention to 
rigorous methodology that is required of primary researchers.- This paper is 
based on a conceptualization of the research review as a scientific inquiry 
involving five stages which parallel those. of primary research: (a) problem 
formulation; (b) data collection; (c) evaluation of. data points; (d) data ana- 



iysisland interpretation, and; -(e) public presentation. 



/ 



*the results of two studies will be reported which experimentally examine 
different facets of literature reviewing. !n Study I, reviewers are asked to 
integrate literatures that vary in the findings and operational homogeneity >of 
their constituent studies. Some reviewers are also asked to make formal judg- 
ments of research quality. These manipulations (along with several .individual 
differences among reviewers) are tested as antecedents to the reviewer's «(a) 
decisions about'jfcested^hypotheses , and (bj recommendations for future, research. t 
Study II manipulates the literature size and findings and tfie reviewers analytic 
Interpretation strategy (statistical versus traditional). The' dependent variables 
are again reviewer perceptions about the hypothesis and 'needed future research. 

The studies examine aspects of research, reviewing that have never before, 
received systematic study. There results should have considerable impact on . 
future discussions about how to. carry out this critical scientific-activity. 
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*. The use^of quantitative procedures in literature reviews has in- 
Creased largely- in: response ! to increases in the numfier of -studies rele- 

vant to' particular research hypptheses. integrative reviewing,- hdwever, 

'/ " ✓ ~~ i , • <• ' * * 

' contains many decision points,'in addition to how to synthesize studies, 

and j all of these have bee'n affected by the expanding -research evidence. 

/ • * 

Critics of meta-analysis fiave pointed , to problems in .other phases ef re- 
viewing caused by large literatures apd have accused the quantitative 
procedures-of either creating or inadequately addressing fhe.se problems. 



In>tact, neither^ the case. The synthesis (ff- studies Js<5nly one of 
'several independent activities 'involved in. literature reviewing^ I would> 
lilje'to present a model of the literature review that (lpconceptualizes. 
'research integration as a scientific process and (2) suggests systematic 
guideline^ for evaluating .the* validity of review outcomes. After describ- 
ing the inodel, examples of reviews that were. attempts to employ the guide- 
lines will be presented. ; , ./ . • . _•«..: 

The Stages of Integrative Research Review ' " ' l, N :[ 

Figure 1, describes 'the integrative review process {Cooper, in. 

* .* * * * 

press). Five stages are' identified by their "particular research, fu^fc-. 

v * 
tion. Sources of variance and potential threats to the validity of re- 

■•_».*•'. »^ > '• 

view outcomes' whi>ch are associated with each stage are , listed. , 
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Problem formulation stage . The first stage in the integrati/e 

review is the problem formulation stage. During problem formulation, ■'. 

the variables involved in the inquiry are defined in two different ways, 

conceptually and operationally- § -? t ^ 

) ■ } + % . I 

Tf\e x t&it source of variance in revijsws enters during concept; . \ 

identification. Two Reviewers using 'an identical label' f&r an abstract 

concept can employ very different operational- definitions or levels of 

■ . . . * \ 

abstraction., Each definition may contain 'some operations excludetf by 

the other, or- one reviewer'? definjtion'^may completely contain the other. 

Multiple -operations for obstructs also affect' review outcomes 
by making it possible for reviewers^ to vary in the attention they pay 
to' methodological distinctions, in the literature. This variation is 
attributable Uo differences i*h the Way operation^ are treated after the 
relevant literature has been retrieved.. Two reviewers employing identi- 
cal conceptual definitions- and reviewing- the same.set.of studies-can 

still reach decidedly different conclusions. If one reviewer retrieved 

• ■ * > . / . 

more method information and recognized a, method-dependent relation that 

• ■.,/»"•" ' * - 

another reviewer did not telt, the two conclusions c^uld be orthogonal 

to one another. \ r ■ . 

'.Each source of « variance introduces a potential threat to Wle va- 
lidity o%a review's .conclusions. First, reviewers who focus "oh. only a 
few empirical 'realizations leave open rival interpretations for thefind 

* « * 

ings. Also, very narrow conceptualizations'provide little information 

/ ' • ' - • . * .' ' v • 

about how'matry: different contexts a finding .applies to. Therefore, re- 

viewe.rs who- employ broad 'conceptual definitions can potentially produce 

. jno^e vaTid % conclusions than reviewers using narrow definitions.- '/M. 
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The word potentially was used because. of the second threat »to va- 
lidity associated" with problem definition. AsPfesb^ (19^8) notes, t 
differences (in studies) are cancelled in the use of very broad categories, 
which'leads to the erroneous 1 conclusion that research results indicate • 
negligible differences in outcoiifes . , . 11 (p. 514)- .We can assume, there- 
fore, that reviewers who examine more operational 'details within their* 

\ . ' u ' ' ' " . \ 

•broacT constructs will probably produce more externally valid conclusions; 
These reviewers present more information about contextual variations 




' that clo and do not 'influence the review outcome, 4 
• \, . , / • \ 

Data collection s,tage - The major decision an inquirer makes dur- 

* ^ * * \ * 
ing thfe data collection" stage involves the population of elements that 

wjlTbe the referent for the inquiry. * . ^ 

- -Identifying populations for research, reviews' isr, complicated "by ' 

\the fact that reviews involve two targets- . First, the inquirer wants 

<$ r * * ♦ 

. the review findings to pertain to all prev4«Qus research on the problem. 

Reviewers can exert some tori^rol over whether this goal is achieved 

\ through thefr choice of information sources- In addition, the reviewer*^ 

. hopes that the included studies Will allow generalizations to the indi- 
/ - \ • ' ' r . ^ / 

viduals that interest the | topic area; The reviewer's influence vis con- 

strained .at this p.oint*byi the types of individuals who were sampled by 
primary Researchers. ' , A ' . ' " - \ 

* • ** *r* - • » ' * 

• ' Discrepancies in review conclusions af*e created by differences 
t in the channels ^reviewers use to retrieve information, such as invisi- v-^ , 
. ble collQejes, -citation indexes and .abstracting 'serviced. \ The studies - ^ 
9 .available . irpugh different sources ' often' a>e* different f r*om 'one ano- 
' ther and Smith (1980) has demonstrated this empirically. . It is, likely, 
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• l * . • ■ ♦ • 

» * - • * 

* . - that two reviewers v*ho use^different Aefhniques to locate studies will % 
end up wfth different evidence and vTTTl^potenti^ny reach different, 
conclusions . f . , - v * ' 

* 9 , * * 

The first threat to data-jjathering vaTidity, theh^ is that the 
review may not include, and probably will not include, all studifes^ppr- 
/tinen-t^tiKjthe topic of interests A revieweKwho has utilized the broad- \ 
est sources of information is most likely to retrieve a set of ; results 

which resembles the entire population of previous research'. . 

' \ ' r * ' 

The second threat to validity occurring during data gathering is • 

- • - ^ .f 

thai the individuals in the VetrieV^'d studies may not represent all in- 

dividuals in the target population. The reviewer cannot be^faulted for 

' . ' the existence of this threat if retrieval^ procedures were exhaustive, 

/ * • \ 

However, reviewers *wh6 qualify conclusions with information about the 

kinds- of people missing or overrepresented in studies probably run less 

* c r\sk of overgeneralfzatton. • ' . 

Data evaluation stage . After* clata is collected, thejj),quirer • 

' makes .critical ^adgments about* the quality of individual data points, • 

Each data. point is examined in light of surrounding evidence to deter- * * 

-.»,».» s * 

mine whether -it* is contaminated by factors irrelevant to. the problem -\ 
under consideration- . • s 



The first Source* of variance introduced during' data 'examination 



\ 



* is created by divergence in reviewers 1 criteria for evaluating the . 

> • ♦ * » 

/ t . ' quality, of research. For instance, Gottfredson <(1978) studied* editors \ 
t '. ' and authors^ in' nine APA journals ancT suggested that inter judge agreement 
on/quali^/- was Relatively modest 1 ' (p. 928). , , 
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\ 
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Another source of variance in review conclusions is the .degree 
to .Which factors other than research (quality -affect evaluative decisions. 
To demonstrate, Mahoney (1$77) found thfet the methods, discussion and 
. contribution. of manuscripts were evaluated more favorably if the study 
confirmed the reviewer's predisposition about the result- 

. - 4 

"The use of any evaluative criteria other than methodological 

V ' - * * 

quality, ought to be considered a tftneart to the internal validity of a 

4 } 

research review As Mahoney states, "To "the extent that researchers 

. . I I 

display (confirmatory)* bias, our adequate understanding of the process^ 
es, and parameters of human adaptation may be seriously jeopardized" . 
(p. 162). ' s 

A second threat to validity durihgevaluation is wholly beyond ^ 

1 1 

the control of the reviewer. This threat involves incomplete reporting 
by primary researchers. If a reviewer must estimate or omitlwhat hap- 
pened in'these studies, wider confidence intervals must be placed around 

i 

rey^ewconclusions. ' ' 

Analysis and interpretation stage . During an- sis and inter- 
pretation, the separate data- paints -collected by the 'inquirer are syn- 
thesized into a unified statement about the. research problem. ^Inter- 
pretation demands that the inquirer distinguish systematic data patterns 
frof* "noise" or chance fluctuation. To carry out this function, the in- 
quirer must apply some rules, of inference. . 

..... "TV . ^ . " . 

Review conclusions' can differ because reviewers 'ei^loy different- 
analytic interpretation techniques. A systematic relation which cannot 

/• • • 

'T * » 

b£ distinguished" from noise tfncJer one set of' rtiTes'ijiay be differentiated 
under another set. • * 
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The first threat to. validity accompanying , the 'analysis and inter- 
pretation stage involves. 'the rule of infe'rence that a reviewer employs. 
Jn non-quantitative reviews, it is' difficult to gauge the appropriated 
ness'of inference rules because they are not very often made explicit. 
For quantitative reviews, the suppositions of statistical tests. are gen- 
erally known and some statistical b.iases in reviews can be removed.' Re- 
gardless of the strategy 'used for analysis- an4 interpretation, the pos- 
sibility a'lways exists 'that the reviewer has used an invalid rule'for 
Inferring a characteristic of the target population." *; 

The second threat" to" validity is the misinterpretation of reviewr 
based evidence as suppbrting statements about causality. In order to 
explain method -generated variance in study outcomes, reviewers will try 
to associate the differences in*results with differences in study pro r 
cedures. While the 1 reviewer- may be tempted to do so, he or she cannot 

9 m * 

rule but .the possibility that the. review-generated relation is spurious". 

i* , ■ • r /| 

Many other variables are confounded with'the original- experimenters ' 

- % ' • 

^choice of a study procedure. f Spurious relations are'possible because 

. the reviewer did not randdmTy assign procedures to experiments. / .* 

Public presentation stage* , Finally, the' production of i public 

•document describing the review is. a £ask with profound implications for 

,the .accumulation of knowlecige^ v . ^ 

■' -Two threats to validity accompany report writing. First, the 

omission" of details about how "the review was conducted reduces the re- 4 

plicability of the review conclusion, Without sufficient .detaiM^the 

reader. is unable t& ascertain whether a^ersona.1 search of the* 1 i ter'a- 



ture would lead to a similar conclusion. 
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The second threat involves^the omission of evidence that^other. 
•Inquirers find important. * Matheson et al. 0?78j observe that "as re- 
search pn a specific behavior progresses, more details cdnc€rn|ng- the " 
experimental conditions are fpund to be r|le'vant" (p.. 26^). .A review 
will quick-ly become obsolete ff it* does not address^the variables and' - 
relation? which are (or 4 will , be). important to an area. 

Examples of Reviews * v ' 

* / The supposition underlying this model of . literature reviewing 
js ;that it is a data-gathering exercise 'which needs to be evaluated- 
against scientific criteria.* 4 As with primary research, reviewers must 
take -precautions to avoid bias in conducting the^' r study . " Equally im- 
portant", the reviewer must produce a' report which allows readers to 

* ** 

assess the review's validity and* to conduct direct replications, if 

they so desire'. Also'similar to primary research, the- perfect litera- 

*• • , - / 

tiire review, does -not exist. Many reviewers have, however, applied the 

extra timer effort and expense needed to produce reviews with consider 

/ 

'ably greater validity and replicability than has traditionally been 

" « - » 

acceptable. My colleagues on today's panel are among these reviewers, 

as are many ptherl who predate the "reviqw-as-research" notion (see 

i 

Glass, McGaw & Smith, '1981; Rosenthal x 1980). In the time, remaining, 
I would .like to briefly describe the -efforts of three of my -students 
who conducted reviews in very different areas using diffipent tech- • 

niques. What the reviews have, in common is that they all attempted to 

* '' ' ' ' >■ • ' ' * 

apply the guidelines described earlier. ^ • 

"(he first .review-was .on the relation between locus of control 

(or a person's belief about whether or not the/ control the, things 
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that .happen* to' them) and academic achievement. This Review was con- 
ducted by Maureen Findley, a grajiuate^ student in*sodal psychology 
(Findley $ Cooper, irb press )♦ Vive previous* reviews of the loftus of 

I • 

control -achievement .relation concluded that' a positive association 
existed between the variables* but the reviews differed in their con- • 
fidence in this conclusion. The reviews also varied in their target 
populations, with some' focu£TTifc| on children, some on adults and, some, 
on all age groups. Finally, the reviews differedMn the mediator? 
they suggested^ might affect the size or existence of the relation.- Of 



the five reviews, tlje most- exhaustive contained 36 empirical studies. 
Maureen's goal was to comprehensively search the literature and exa- 
mine all target populatioas and suggested- mediators in* a single review. 
* " .Three databases, .Psychological Abstracts, ERIC, and Dissertation 
Abstracts International,- were. searched by computer. The index words 

"achievement" or "performance" were crossed with "locus. of control" or / 

~~ • / 

"internal-external." Eight hundred and two studies were located which 
* * - . % 

•either mentioned these terms jn their title or abstract or were so 
classified by -a person who read the, entire document, - Th?' titles and ab- 
stracts ofHhese, studies were provided by the computer and these were 
usfctf"to ^educp the number of potentially relevant reports.. Ultimately, 
208 studies, were examined' in their entirety clnd 98 relevant studies . 
were.founxi, nearly three times- as many as the next most exhaustive re- * 
view. The 98 studies contained 275' tests of the hypothesis. 

Across all. studies, the average correlation Between locus of - 
control and achievement was r ~ +.18. This combined result would rer * 

'quire over 3,000 unretrieved, null-summing studies to be -reversed at 
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the p < .05 level of Significance (see Cooper, 1979). Table 1 provides 
a breakdown of these- results according to. the' six mediating variab-les. I § 

most frequently suggested by previous reviewers. Jhe analyses of dlf- 

. ■ \ ' - , , • ^ 

ferences in correlations across studies revealed that male samples pro- 
' duc^d stronger, relations th§n female samples and £ha1^ junior high school 
,-students produced stronger relations than either elementary school^ or 

college students. ' In addition-,' fneasures of locus of control specific 

to academic outcomes -teh^gd to- show stronger associations, with achieve- 
.inent than more "general locus of control measures a^d stronger relfi&ons • 

wefe 'found in studies employing" standardized as opposed to informal 

assessment? of achievement. » 



- • • 'Place Table 1 .about here 

^ " Maureen's discussion of the results was ablfe to evaluate the * 
magnitude *of the* ♦locus of control relation and»pay particular pttentiop 
to confTTctTln the results of the previous reviews. - . , / 
* ' M<en'Ottenbacher, a graduate student in special education, con- - N 
ducted a review of studies testing-the effectiveness of. drya treatments 
of hyperactivity, in children. Ken was able to* locate Bl*. stupes .that \ 
met very stringent- criteria. All 61 studies employed two'rgroup compar- 
isons between a drug condition, a no treatment^ontrol condition or a 
placebo condition, in additidn, all studies used random -assignmeirTof - 
children to. conditions and a double*, blind procedure in admi-^-tering 
the treatment -dnd recording the dependent variable. • . 

Table -2 presents a stem-^nd-leaf display of the 408" separate ~ 
^indexes uncovered by the literature search. Before synthesis, Hedges' 
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'.A • - . N < 

correction factor was employed. Most interesting in this review was 

* •• ■ , 

the' comparison' of the effects of placebo and drug treatments. The mean 
d-index f 0/ compari sons of drug treatments versus no treatment control • 
groups wa?+l,.21. Placebo groups' versus no treatment controls pro- 
ducefl a mean effect Bf d = +.19 while drug versus placebo comparisons 
revealed a mean dr-index of +.69. This analysis led Vo a conclusion 
that about 3/4 of the drug effect was probably due to the 'drug itself 
and 1/4 due to the expectancies that surround-drug therapies. Ken al- 
so-found that stimulant drugs were more effective than rionstimulahts 
in reducing hyperactivity and that the effect of 'drug therapy was un- . 
related to the age or I.Q, of the child. or to how hyperactivity was 
measured. % ' 



Place Table 2 about here 



1 _ 1ir , « 

^ A third review Was condubted by Julie Yu, a graduate student in, 
Marketing. Julfe : was interested in how response rates to questionnaires 
.were affected by the research design. She examined over a- dozen di f- 
ferent techniques that' surveye^s use to increase whether or not an in- 
dividual agrees to complete a questionnaire. ~ 

- The unique aspect of Julie's .task was that all .studies employed 
identical dependent variables, namely .the percentage. Of contacted in r - 
dividual* who agree to respond- . Thus,' rather than working with. study . 
.probabilities and effect sizes,, it was possible' to directly ^ombine raw 
data. Literature searches of BRS/Inform, Management Contents,, Psychol 0- 
gical Abstracts and the Social Science Citation Index uncovered 25 re- 
levant .studies; and; 60' more were found through a manual search of refer- 
ences in bibliographies'. • 
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* Tattte 3 examines the effect of a monetary incentive on response 
rate. For each condition^ a weighted average response rate i'$ given 
along with the number of contacts and separate response rates the aver- 
age rate is based on. The standard deviation' of the rates is/fclso pre- 
sented. Four rates were differentiated^ Experimental and cotjtrol 
rates are based on studies that explicitly manipulated the presence or 
absence, of monetary .incentives. ' The without control rate Is 'based on 
studies in which all participants received «in incentive and the absent 
rate is based t>n studies in which no participant was paid. 



Place TabT* .3&about here 



A chi-square statistic fp^the Uperiment versus control fre 7 
quencies of responding was highly significant with an associated phi-. *.' 
coefficient of +.15. A descriptive correlation was also generated by 
pairing response rates with the ^amount of money offered. Thi^ corre- 
lation equalled +.61, indicating greater monetary .incentives' led to • 
higher responding. These analysis procedures" were applied to each 
technique. "Julie found significantly higher' response rates associated, 
with both prepaid and promised. monetary incentives, nonmonetary rewards,, 
preliminary notification; personalization of the request, and fo^ow-up 
contacts. »The effects of a. cover letter, assurances of anonymity, pro- 
vlding a deadline, and providing return postage were a*17 nonsignificant'. 

Conclusion ' ' 
> 

Obviously, this'discussion has not-done justice to "the detail 
and complexity of these reviews, but I hope the general' point is c^-ear/' 

j ____ m • 

More .scientific guidelines for conducting integrative research reviews 

. •: - .. } ' ' * v 
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are not only desirable but they .are feasible to apply. Alsp, rigorous 
criteria, Will fibt produce reviews that, are uncreative or^mecbanical^in 
nature. The expertise arfd intuition of the reviewer will tie challenged^ 
to capitalize on 'th& opportunities, for mining information, unique to 
each problem area. . Scientific reviews, however, should have much 
greater potential for creatine consensus, among scholars for focusing 
discussion on' specific and testable areas of di sagreemejj^^JBT 
*flict does exist.,'. \ '* 



N 
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The Literature Review Conceptualized as a R^searck Rroject 

'Stage of Resfeardh * • * } * 



Stage ■* 



Char ac ter i s ti c s 



Problem Formulation 
— v 



Data Collection 



Data Evaluation 



Analysis and <r\ 
Interpretation I 



Public 
Presentation 



Research Question 
Asked-" 



What evidence should be 
included in the revffew? 



Primary, Function 
in Review 



Constructing defini- 
tions that distinguish 
relevant from irrelevant 
s^W^ies . 



Procedural Differ- 
ences Which Create 
Variation in Review. 
Conclusions * 



Sources of Poten- 
tial Invalidity „ 



in Reviey 
Cone 1 us 10: 




1. Differences in 
abstractness of r 
deT initi'on. 

2. 'Ddffer'ejfces in ' 
operational detail. 

■ 1 . Narrow concepts may 
make review conclusions 
less general, 

2. Superficial op.er.a- # 
tional detail may 
obscure interacting «« 
variables. 



What procedures 
should be used to 
find relevant 
evidence? ' ; 



Determining which 
sources of ~potefvt: 
tially } Relevant 
studies to examine. 



Differences in 
the research con- 
tained in. source^ 
of information. 



1. Accessed- studies 
may be qualitative- 
ly different from 
the target; pAular- 
tiori of studies. 



What retrieved 
evidence should, be 
included in the 
review? 

Applying criteria 
to separate "valid 11 
from "invalid 11 
studies. • m 



1 .. Differences in 
quality criteria. 

2. Differences in 
the influence of • 
nonquality criteria. 

1. Nonquality fac- 
• tors may cause im- 
proper weighting 
Sf study informa- 
tion. 



2. People sampled 
in accessible stud- 
ies may be, differ- 
ent* from, target pop- unreliably. , 
{ilation of people-, ' 



2. Omission^ in 
study reports may 
make conclusions 



WJiat procedures 
should b« uspd to 
make inferences 
about t\ae liter- 
ature as a who Ire? 

Synthesizing 
valid retrieved 
studies . 



Differences in 
rule is of infer- 
ence. 



What information 
should be' included 
in the review 
report? 



Applying editorial 
criteria to . x 

separate important 
from unimportant 
information. f 

Differenced in 
guidelines for 
editorial judgment. 



1. Rules for dis- 
tinguishing pat- 
terns from noise 
may be inappro- 
priate. 

2 . Review-based 
evidence may be 
used t^i^fer 
causality. 



1. Omission of 
review procedures i 
may make conclu- 
sions ir reproduct- 
ible. 

2.. Omission of * 
review findings - 
and study procedures 
may make conclusions' 
obsolete. • • M * v ? 



Scientific Guidelines . 



• * Table 1 

Average Effcect Size £or Subgroupings of 
- . Study Characteristics . 



Characteristics 



Average CorTelat ions 



SD 



Cender . 
Males * ' 
Females 

>* 

Age 
.College 
High school 
Junior high 
4th-6fh . 
lst-3rd grade 

Race 

* Black 
White 

Socioeconomic Status 
Middle class 

* Lower class 

(ocus of Control Measure 
General 
Specific 

Achievement Measures 
Classroom-related 
Standardized Achievement 
Standardized, Intelligent 




+ .20 
+ .11 



+ .23 
+ .35 
+ .24 
+ .04 



+ .25 
+ .25 



+.26 
^-.35 



+.18 
+.30 



+.16 
+.21 
+ .24 



.14 
.18 

\»5 
\ .to' 

' .22 
*.15 
.06 



.47 
.17 



.11 
.34 



- .16 
.22 



.15 
.17 * 
.45 



27 
18 



32 

8" 

7 
21 

4 



3 
8 



S 
- 4 



15' 
12 



\ 



45 

36" 

ft 



* ' ^ 

Note i. N. iSothe number" of studies upon which the average correlation 
and SD are based. 



r v» «n#iw *m* anrf-CooDe^ H. The "relation between locus of control and 
From: ^^"dXooper-, and Soo1a1 p sych o, ogy . 



i5 



.20 



^37^ Stientitip UuideHnes 
Table 2 . t 



/>tem 



, 2;1 
.2.0 ♦ 

1.9 
1.8. , 
'1.7 
1.6 
1.5 
*1.4 

• 1.3 
' 1.2 . 

1.1 

1.6' 

.9 

*8 
.7 . 
.6 

' -.4 

... .3 ' 
' * .2 

' .1 '.' 

• .0 



* * d-indexes for tho comparisons of &Vrug versus control/, drug 

— . • , .* 

"versus placebo, and placebo versus control. 

. •»-•«!* •* 

Dru» vs. Control Drug vs Placebo Placebo vs Control " Total 



'1 



5. 

4- 

* 0 ; * 

5 ' 
8 . 
2 

16" 

-39 - 
A 

\ 

09* , 

' I 

46 . 



v 



8 



•2U 
1 ' 

0688 \ . ■ 
,126 I . 
62 ' 

8 

1 

8 , > 
12399 

05* 

>i 

■ 1125779 
"07 i' 

2489 . J 
:-l68 ; ' 
V.478 
56 , . 
005v 



\ \ 
6. 

3. • 

9 1' 



0' 



8 



00?)DO, 



5; • 
8 



02^ • • 
1 

' 05688 
12368 

o^Tn 
168 

68 V 

12333999 
0549 \ 
59 

1125779 
00079 
2489 • 

* 

•; 146688, 
*78 ■ 

■5.6- . . 
' 00000005 



\ 



Maximum. 



?3 



Medijsm^ 

Minimum) 
Mean. ' 
SD 



•*2.77 
1.55 
1.10 

.59 

.34 
1.'21 

.67 



2.08- 
1.30. 
.'.69 ' 

.42-- 
• . .00 r 

.34 

' « 

: i3L 



.93 ■ 
.19 ; 
.00 . 

-1.30 

.52 . - 
» .72 1 



^2.77 
T.30 
>\ ...80 

' .- 38 r 
.^1.30 

.84 

.60 



• 'Note: 1 Two value's, 2.' 77 and -I^P^.are not included in tire table. 
& "Hi fromi Ottenbacher, K; and Cooper, J. Drug. v treatment' of hyperactivity^ 



Manuscript 



r Sc1eKtiffc v Guide1ines 



•J 



Table 3 



Effect of Monetary Incentives on'Response Rate'; 




From: Yu,'J. and Cooper, H. A literature review of research design effects on * . ^ 
response rates to questionnaires* . Manuscript under review, .1981.. ■■•«V 



,6 ^ ^ 



1 ' 



*«v c ?. - 22* ■ * * 



